1. Field of the Invention
The present invention relates to multi-modal interfaces and more specifically to a multi modal interface application for recording and accessing medical information.
2. Discussion of Related Art
Multimedia interactions are being applied to various industries to enhance the exchange of information. One example of this trend is the medical field. Research and development continues on such systems as the “MAGIC” system from Columbia University (MAGIC: Multimedia Abstract Generation for Intensive Care) that uses multi-modal techniques for providing a summary to the intensive care unit after a patient has surgery. The goal of the MAGIC system is to provide a set of tools for producing multimedia briefings that meet the needs of a variety of caregivers, such as specialists and nurses. The MAGIC system employees natural language generation, knowledge-based graphics generation, and knowledge representation and reasoning systems. See M. Dalal, S. Feiner, K. McKeown, D. Jordan, B. Allen, and Y. Al-Safadi, “MAGIC: An Experimental System for Generating Multimedia Briefings About Post-Bypass Patient Status,” AMIA Fall Symposium, Washington D.C., October 1996.
FIG. 1 illustrates the MAGIC system architecture 100. The medical databases 102 provide the basic information for the patients. The data server and filter 104 accesses the several medical databases 102 for collecting information about the patient's condition. The databases 102 accessed by the data server 104 include LifeLog database system and a patient clinical database. The data filter 104 selects relevant and important parts of this data and several built-in patient-independent hierarchies (including a Domain Hierarchy and a Concept Hierarchy), infers some new information, and creates a Patient Data Hierarchy used by all subsequent components to represent them.
The general content planner 106 receives the data from the data server and filter 104 and uses the Patient Data Hierarchy and plan library build a Presentation Plan that expresses the high-level Communicative Goals to be accomplished by the Multimedia Briefing. A media allocator 108 chooses one or more media to express each communicative goal in the Presentation Plan. A Medium-Specific Content Planner can expand the Presentation Plan by building detailed plans for the communicative goals assigned to its medium. A Medium-Specific Media Generator uses these detailed plans to generate its portion of the briefing.
The Media Coordinator 110 ensures that the Planners and Generators of different media are consistent with each other. The Media Conductor 116 takes the entire presentation plan and “play” it so that various media are coordinated together, and a single coherent multimedia briefing results. The Constraint Solver 122 provides the central facility for representing the constraints among the different parts of the Presentation Plan and ensures they are mutually consistent.
User Models 124 will represent preferences of individual users and groups of users about the content and format of the Multimedia Briefing. Speech content planner 112, speech generator 114, and graphics content planner 118 and graphics generator 120 are coordinated via the media coordinator 110 and media conductor 116 to present the multimedia medical report for the ICU.
FIG. 2 provides an example of exactly how the MAGIC system presents a patient report. The patient report may be provided to a doctor or a nurse and a button option is presented so that the user can obtain the tailored patient information.
The MAGIC system presents information in a graphical form. FIG. 2 presents the graphical portion of a multimedia report for a “S. Jones.” Basic patient information is presented in a heading 202. A three dimensional image of a patient is presented 204. As the report begins, synthetic speech audibly presents the basic information and this portion of the screen is highlighted to let the user (nurse) know that the patient background material is being spoken.
Next, via the synthetic spoken instructions, the MAGIC system gives the patient medical condition and treatment. While the audio is playing, a coordinated presentation of graphics is provided. For example, a text box 210 pops up while the system speaks the composition of the drops 208 given. The text “Swan-Ganz with Cordis” 206 describes a treatment and location on the patient 204 of the treatment. Other text such as “Peripheral IV” 212, 220, “Boles” 214, “Blood” 216, and “Arterial Line” 218 describes further treatment. As the audio portion continues, other pop-up boxes associated with the audio and located near the area of treatment aid in the overall multimedia presentation.
While the MAGIC system presents a helpful multimedia presentation of medical information, the system nevertheless has deficiencies. For example, the MAGIC system receives information from the various databases to generate its presentation. As mentioned above, MAGIC receives data from sources such as a data server, medical databases and patient databases. Other than the normal methods where medical professionals input data to databases, MAGIC provides very little, if any, means of user input prior to the multimedia presentation. Furthermore, it does not appear that there is any opportunity for user input or interaction with the system during the presentation. Therefore, if the information presented to the user (nurse or doctor) is incomplete or the user desires further information—there is no means for interacting effectively with the system.
Physicians and other medical personnel have to quickly record and access large amounts of medical information regarding a patient's condition and the treatments they have received. The medical professionals use this information to record what has happened, to guide further diagnosis and treatment, and also to serve as the basis for billing for treatments and procedures carried out. Currently, the primary method for recording this information is by handwritten shorthand on paper. Some commercial solutions exist which provide the doctor with various forms to fill out in order to indicate the patient's condition. These forms are typically text-based forms. There are also commercial services available where the doctor can call into a central voicemail system and dictate the report of what happened to the patient and the treatments applied. A service provider then transcribes the dictated material and makes it available to the physician. In other situations, when the report is handwritten, often someone else is paid to decode the shorthand specification of the condition and treatment and determine the appropriate billing codes.